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Congressional Requesters: 

Subject: Homeland Security Grants: Observations on Process DHS Used to Allocate Funds to Selected 
Urban Areas 

In fiscal year 2006, the Department of Homeland Security (DHS) provided approximately $1.7 billion in 
federal funding to states, localities, and territories through its Homeland Security Grant Program (HSGP) 
to prevent, protect against, respond to, and recover from acts of terrorism or other catastrophic events. 
The Urban Areas Security Initiative (UASI) is a discretionary grant under this program, and since fiscal 
year 2003, Congress has directed DHS to target UASI funding to high-threat, high-density urban areas to 
assist in building capacity. 1 To meet this requirement and inform funding decisions, DHS developed a 
method to estimate the relative risk of terrorist attacks to urban areas. From fiscal year 2003 through 
2005, DHS used a number of risk indicators such as population density and threat to allocate UASI funds. 
UASI funding increased during this period from about $96 million to $830 million, while the number of 
urban areas that received grants grew from 7 to 43. In fiscal year 2006, DHS awarded approximately 
$711 million in UASI grants — a 14 percent reduction in funds from the previous year — while the number 
of eligible urban areas identified by the risk assessment decreased to 35. For fiscal year 2006, DHS made 
several changes to the grant allocation process, including modifying its risk assessment methodology, 
introducing an assessment of the anticipated effectiveness of investments, and combining the outcomes 
of these two assessments to inform funding decisions. 

The results of the UASI eligibility and funding allocations in fiscal year 2006 raised congressional 
questions and concerns about DHS's methods in making UASI determinations. Several congressional 
members requested that we examine aspects of DHS's UASI funding process, and the fiscal year 2007 
DHS Appropriations Act directed us to examine the validity, relevance, reliability, timeliness, and 
availability of the risk factors (including threat, vulnerability, and consequence) used by the Secretary of 
Homeland Security for the purpose of allocating discretionary grants. 2 On November 17, 2006, we 
responded to the mandate and the request by briefing congressional staff on the results of this review 
(see app. I). We specifically examined (1) DHS's method of estimating relative risk of terrorism in fiscal 
year 2006; (2) DHS's process for assessing the effectiveness of the various risk mitigation investments 
submitted in UASI applications; (3) how DHS used estimated relative risk scores and assessments of 
effectiveness to allocate UASI grant funds in fiscal year 2006; and (4) what changes, if any, DHS plans to 
make in its UASI award determination process for fiscal year 2007. This letter and the accompanying 
appendices transmit the information provided during those briefings. 



'Prior to fiscal year 2003, funding to urban areas was provided under the Nunn-Lugar-Domenici Domestic Preparedness 
Program, which was administered by the Department of Defense starting in fiscal year 1997, and later the Department of Justice 
during fiscal years 2001 and 2002. Other grants under the HSGP include the State Homeland Security Program, Law 
Enforcement Terrorism Prevention Program, and Citizen Corps Program, among others. 

2 Pub. L. No. 109-295, 120 Stat. 1355, 1370 (2006). 
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To gain understanding and describe DHS's process for awarding fiscal year 2006 UASI funds, including 
eligibility and award amount determinations, we reviewed available documentation and interviewed 
knowledgeable officials. During this document review and our interviews, we gathered information 
about the data DHS used to analyze relative risk and what efforts it had in place to ensure data 
reliability. For example, we collected information on DHS's consultation with states to obtain and 
review critical infrastructure data and DHS's internal assessments of intelligence data. Additionally, we 
examined DHS guidance and methods for implementing an assessment of effectiveness of applicants' 
plans to mitigate risk. Finally, during our review of documents and interviews with DHS officials, we 
also collected information regarding any changes to the determination process for the fiscal year 2007 
grant cycle. We conducted our work from September 2006 through November 2006 in accordance with 
generally accepted government auditing standards. 

Summary of DHS's Process for Allocating UASI Grant Funds in Fiscal Years 2006 
and 2007 

The Risk Assessment. In fiscal year 2006, DHS used its risk assessment to identify urban areas that 
faced the greatest potential risk, which made them eligible to apply for the UASI grant, and based the 
amount of awards to all eligible areas primarily on the outcomes of the risk assessment and a new 
effectiveness assessment. DHS enhanced its risk assessment by including three components — threat, 
vulnerability, and consequences — to estimate the relative risk of successful terrorist attacks to urban 
areas. The risk assessment was used to inform DHS's selection of eligible urban areas. DHS also 
implemented a competitive process to evaluate the anticipated effectiveness of proposed investments to 
address homeland security needs by using peer reviewers, who were homeland security professionals 
from fields such as law enforcement and fire service. The peer reviewers scored the investments using 
criteria, such as regionalization, sustainability, and impact. According to DHS, it combined the outcomes 
of the risk and effectiveness assessments to inform the funding allocation decisions in fiscal year 2006, 
but the Secretary of Homeland Security made the final UASI grant decisions. Officials also reported no 
significant changes to the risk assessment process for next year's grant cycle, but other decisions, such 
as the identification of eligible urban areas through the risk assessment and how much weight risk and 
effectiveness will be given in determining amounts, have yet to be made. Figure 1 illustrates UASI grant 
determination process in fiscal year 2006. 
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Figure 1 : Overview of DHS's UASI Grant Determination Process in Fiscal Year 2006 
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Source: GAO analysis of DHS documents and information provided in interviews. 



In fiscal year 2006, DHS estimated the risk faced by urban areas by assessing the relative risk of 
terrorism 3 as a product of three components: (1) threat, or the likelihood that a type of attack might be 
attempted; (2) vulnerability, or the likelihood of a successful attack using a particular attack scenario; 
and (3) consequence, or the potential impact of a particular attack. To estimate the relative risk, DHS 
assessed risk from two perspectives, asset-based and geographic, then combined the assessments 
weighting geographic risk twice as much as asset-based risk. According to DHS officials, it made the 
judgment to weight geographic risk 1.0 and asset-based risk 0.5, since a potential loss of lives within an 
area would contribute to how geographic risk is assessed. To estimate asset risk, DHS computed the 
product of threat, vulnerability, and consequence by assessing the intent and capabilities of an adversary 
to successfully attack an asset type, such as a chemical plant, dam, or commercial airport, using one of 
14 different attack scenarios (e.g., nuclear explosion or vehicle-borne improvised explosive device). 
Simultaneously, DHS assessed geographic risk by approximating the threat, vulnerability, and 
consequences considering general geographic characteristics mostly independent of the area's assets 
using counts of data such as reports of suspicious incidents, the number of visitors from countries of 
interest, and population. In DHS's view, the two estimates of risk — asset-based and geographic — are 
complementary and provide a "micro- and macro-" perspective of risk, respectively. In calculating these 
relative risk scores and addressing the uncertainties in estimating relative risk, policy and analytic 
judgments were required. For example, DHS made judgments about how to weight asset and geographic 
risk, how to identify the urban boundaries it used to estimate risk, and what data were sufficient to use 
in its risk estimates. DHS used this risk assessment to identify the eligibility cut point, which determined 
the number of urban areas that could apply for UASI funding in fiscal year 2006 and defined high-risk 
urban areas. According to DHS officials, the DHS Secretary selected a point that resulted in 35 eligible 
urban areas, which accounted for 85 percent of total related risk. DHS then decided to extend eligibility 
to 11 sustainment areas that participated in the program in fiscal year 2005, but were not identified in 



3 By using a relative risk value, DHS assessed the risk of potential terrorist attacks to one urban area relative to another urban 
area. DHS estimated relative risk as an ordinal number, which typically is understood to indicate rank order. Further, the 
"distance" between the numbers has no meaning. According to DHS, a classical probabilistic risk assessment, in which risk is 
calculated using historical statistical data to quantitatively describe the likelihood of a particular event (usually expressed as a 
value between and 1), cannot be used, because there are little available historical statistical data to describe terrorism risk. 
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fiscal year 2006 through the risk assessment. 4 Appendix II contains more detail about the risk 
assessment process and describes how DHS used these estimates to determine which urban areas were 
eligible to apply for fiscal year 2006 UASI grants. Appendix III provides information on the data sources 
used in DHS's fiscal year 2006 risk estimates. 

The Effectiveness Assessment. For the first time since the inception of the program, DHS required 
urban areas to submit investment justifications as part of their grant application, so it could assess the 
anticipated effectiveness of the various risk mitigation investments urban areas proposed. The 
investment justifications included up to 15 "investments" or proposed solutions to address homeland 
security needs, identified by the states and urban areas through their strategic planning process. DHS 
used peer reviewers to assess the investments submitted by the 46 urban areas — 35 eligible through the 
risk assessment and 11 sustainment areas. DHS and the states collaborated to identify and select these 
peer reviewers who were homeland security professionals and managers from disciplines such as law 
enforcement, fire service, and emergency communications. According to DHS, it arranged 17 peer 
review panels that included reviewers from a variety of professions, all levels of government, and 
representatives from different regions of the country and from both large- and small-population states. 
These reviewers evaluated, discussed, and scored the urban areas' investment justifications, initially on 
an individual basis, then in panels. The criteria reviewers used to score the investment justifications 
included the following categories: relevance to the interim National Preparedness Goal and to state and 
local homeland security plans, anticipated impact, sustainability, regionalism, and the implementation of 
each proposed investment. Reviewers on each panel assigned scores for six investment justifications, 
which according to DHS officials were averaged to determine a final effectiveness score for each urban 
area. Appendix IV provides additional details about the approach DHS used to assess effectiveness in 
fiscal year 2006. 

Final Allocation Decisions. Finally, DHS used a new method to help determine UASI allocation 
amounts for the 46 eligible urban areas, based primarily on the risk and effectiveness assessments, but 
final allocation decisions were made by the Secretary of Homeland Security. The risk and effectiveness 
scores did not automatically translate into funding amounts, but rather, the scores informed the 
decisions, according to DHS. While all eligible urban areas that applied for UASI grants would receiving 
funding, DHS had to prioritize how funds would be allocated. DHS prioritized those areas estimated to 
have the highest risk of a successful terrorist attack, while still rewarding those areas that proposed 
ways to address homeland security needs that were anticipated to be effective. DHS used the combined 
scores to assign the 46 eligible urban areas into four categories: Category I — higher risk, higher 
effectiveness; Category II — higher risk, lower effectiveness; Category III — lower risk, higher 
effectiveness; and Category IV — lower risk, lower effectiveness. According to DHS, it considered many 
different distributions of funding to each of the 4 categories. DHS officials said that they made the 
decision to give Category I the highest funding priority and Category IV the lowest funding priority. Once 
the amounts for each category were decided, DHS used a formula to determine the grant award for each 
urban area, giving the risk score a weight of 2/3 and the effectiveness score a weight of 1/3. According to 
DHS, these weights reflect its decision to prioritize risk over effectiveness. DHS officials reported 
presenting funding options to the Secretary of Homeland Security, who made the final decision about 
funding allocations. The final funding decision resulted in 70 percent of UASI funding going to "higher 
risk" candidates in Categories I and II. Figure 1 illustrates these funding priorities, as described by DHS 
officials, in which each circle represents a hypothetical urban area and the size of the circle corresponds 



4 According to DHS, extending eligibility to the 11 urban areas reflected feedback from stakeholders on the importance of 
providing funding across fiscal years. In addition, in DHS's view, this decision provided greater transparency in the process and 
fostered long-term planning for program participants. DHS has also stated that any urban area not identified as eligible through 
the risk analysis process for two consecutive grant cycles will not be eligible for continued UASI funding. 
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to the relative amounts of the grant awards. 5 Appendix V provides additional details on the allocation 
method used in fiscal year 2006. 



Figure 2: DHS Allocation Tool for Fiscal Year 2006 UASI Funding 




O Circles are sized relative 
to allocation amount 

I. H ig he r ri sk/h ig he r eft active n ess 

II. Higher risk/lower effectiveness 

III. Lower risk/higher effectiveness 

IV. Lower risk/lower effectiveness 



Source: GAO analysis of DHS documents and information provided in interviews. 



The Fiscal Year 2007 Process 

The fiscal year 2007 process, as described by DHS officials, represents a continuing evolution in DHS's 
approach to its risk methodology for grant allocation. DHS officials said they will to continue to use the 
risk and effectiveness assessments to inform final funding decisions. For fiscal year 2007, DHS officials 
described changes that simplified the risk methodology, integrating the separate analyses for asset-based 
and geographic-based risk, and included more sensitivity analysis in determining what the final results of 
its risk analysis should be. DHS officials said the primary goal was to make the process more transparent 
and more easily understood, focusing on key variables and incorporating comments from a variety of 
stakeholders regarding the fiscal year 2006 process. For the 2007 grant cycle, DHS no longer estimated 
asset-based and geographic risk separately, considered most areas of the country equally vulnerable to a 
terrorist attack, given freedom of movement within the nation, and focused on the seriousness of the 
consequences of a successful terrorist attack. As shown in figure 3, the maximum risk score possible for 
a given area was 100. Threat to people and places accounted for a maximum of 20 points, and 
vulnerability and consequences for a maximum 80 points. In the fiscal year 2007 process the intelligence 
community for the first time assessed threat information for multiple years (generally, from September 
11, 2001 forward) for all candidate urban areas and gave the Office of Grants and Training a list that 
grouped the 168 areas into one of four tiers. Tier I included those at highest threat, relatively to the other 
areas, and tier IV included those at lowest threat relative to the others. 



"The figure does not represent actual urban areas or grant award amounts. 
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Figure 3: DHS Risk Assessment Methodology for Fiscal Year 2007 UASI Funding 
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According to DHS officials, the greatest concern was the impact of an attack on people, including the 
economic and health impacts of an attack. Also of concern was the quantity and nature of critical 
infrastructure within each of the 168 urban areas assessed. DHS reported that the threat information 
used for risk estimates was based upon an analysis of all credible intelligence data. DHS's Office of 
Intelligence and Analysis performed this review and provided the Office of Grants and Training with 
threat assessments and corresponding threat values for each urban area. In contrast, for the 2006 grant 
cycle, DHS used total counts of threats and suspicious incidents and incorporated these into its model. 
In addition, estimates of asset-based vulnerability were assigned values on a cardinal scale of 1 to 
100 rather than an ordinal scale of 1 to 3, which DHS officials believe provided insight into the 
differences between asset types with different values. 

In assessing threat, vulnerability and consequences, DHS specifically wanted to capture key land and 
sea points of entry into the United States and the location of defense industrial base facilities and 
nationally critical infrastructure facilities. The approximately 2, 100 nationally critical infrastructure 
assets included in the risk assessment were selected on the basis of analysis by DHS infrastructure 
protection analysts, sector specific federal agencies, and the states. According to DHS, these 2,100 assets 
include some 129 defense industrial base assets. Assets were grouped into two tiers: (1) those that if 
attacked could cause major national or regional impacts similar to those from Hurricane Katrina or 9/11; 
and (2) highly consequential assets with potential national or regional impacts if attacked. Tier II 
includes about 660 assets identified by state partners and validated by sector specific agencies. On the 
basis of Office of Infrastructure Protection analysis, Tier I assets were weighted using an average value 
three times as great as Tier II assets. According to DHS officials, defense industrial base assets were 
included in the national security index and all other assets in the national infrastructure index. 

Throughout this process, a number of policy judgments were necessary, including what variables to 
include in the assessment, the points to be assigned to each major variable (e.g., threat, the population 
index, economic index, national infrastructure index, and the national security index) with an eye 
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towards how these judgments affected outcomes. DHS officials noted that such judgments were the 
subject of extensive discussions, including among high-level officials. In addition, DHS officials said that 
they conducted more sensitivity analyses than was possible in the fiscal year 2006 process. DHS officials 
noted that because expert judgment was applied to the data, fewer variables were used in the current 
model, making it possible to track the effect of different assumptions and values on the ranking of 
individual urban areas. 

Finally, DHS officials said that the effectiveness assessment process will be consistent with last year's 
process, although a number of enhancements have been made based on feedback received. However, 
no final decision has been made on the weights to be given to risk and effectiveness for the allocation of 
the fiscal year 2007 grants, according to DHS officials. One modification to the effectiveness assessment 
will provide urban areas the opportunity to include investments that involve multiple regions. This can 
potentially earn an extra 5 percent to 8 percent on their final score. In addition, DHS will convene a 
separate peer review panel to assess proposed investments for these multi-regional investments. DHS 
has also offered applicants a mid-year review where applicants can submit their draft proposals to DHS 
to obtain comments, guidance or address questions that the grant may pose (such as little or unclear 
information on the anticipated impact of the investment on preparedness). As in the 2006 process, DHS 
officials have said that they can not assess how effective these investments, once made, are in mitigating 
risk. 

Observations 

Determining an appropriate methodology to assess terrorism risk is challenging, given uncertainties 
such as the limited data on actual attacks and understanding the capabilities, intentions, and adaptability 
of terrorists. The inherent uncertainties in estimating risk require policy and analytic judgments. Other 
federal agencies, terrorism risk researchers, and we agree that threat, vulnerability, and consequences of 
an attack should be incorporated into terrorism risk assessments. DHS has adopted an overall risk 
assessment approach that consists of these three risk factors, and in implementing this approach has 
made judgments in an attempt to address inherent uncertainties. According to DHS's Under Secretary, 
DHS has made significant progress in developing its risk assessment methods, which includes using a 
model based on the three risk factors and incorporating state and local input. However, for the 2006 risk 
assessment process, DHS officials told us that DHS had limited knowledge of how changes to its risk 
assessment methods, such as adding asset types and using additional or different data sources, affected 
its risk estimates. According to a senior technical advisor in DHS's Risk Management Division, DHS did 
not have the resources to undertake such analyses. Consequently, DHS could not assess the effects of 
these changes on risk rankings and the determination of eligibility for, and amount of, UASI grants. This 
official acknowledged the importance of judgments in assessing risk of terrorism and eligibility 
outcomes. 

DHS had a limited understanding of the effects of the judgments made in estimating risk that influenced 
eligibility and allocation outcomes in fiscal year 2006. DHS leadership can make more informed policy 
decisions if they are provided with alternative risk estimates and funding allocations resulting from 
analyses of varying data, judgments, and assumptions. The Office of Management and Budget (OMB) 
offers guidelines for treatment of uncertainty in a number of applications, including the analysis of 
government investments and programs. These guidelines call for the use of sensitivity analysis to gauge 
what effects key sources of uncertainty have on outcomes. According to OMB, assumptions should be 
varied and outcomes recomputed to determine how sensitive analytical results are to such changes. 6 By 
applying these guidelines decision makers are better informed about how sensitive outcomes are to key 



6 Office of Management and Budget, Circular A-94: Guidelines and Discount Rates for Benefit-Cost Analysis of Federal 
Programs, (Washington, D.C.; October 29, 1992) p.10-11 
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sources of uncertainty. While DHS has indicated that it performed some sensitivity analyses for fiscal 
year 2006, it has not provided us with details on the extent of these analyses, how they were used, or 
how much they cost. DHS officials told us they had conducted more extensive sensitivity analyses for 
the fiscal year 2007 risk assessment, but we have no documentation on what analyses were conducted, 
how they were conducted, or how they were used and affected the final risk assessment scores and 
relative rankings. 

Agency Comments 

We provided a draft of this report to DHS for review and comment. DHS provided us technical 
comments that we incorporated into our report where appropriate. 

GAO Contact 

We are sending copies of this correspondence to the requesters listed below, the appropriate 
congressional committees, and the Secretary of Homeland Security. 

Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last 
page of this report. For further information about this report, please contact William Jenkins, Jr., 
Director, GAO Homeland Security and Justice Issues Team, at (202)-512-8777 or atjenkinswo@gao.gov. 
GAO staff members who were major contributors to this report are listed in appendix VI. 




William Jenkins, Jr., Director, 

Homeland Security and Justice Issues Team 
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List of Congressional Addressees: 



The Honorable Robert C. Byrd 
Chairman 

The Honorable Thad Cochran 
Ranking Minority Member 
Committee on Appropriations 
United States Senate 

The Honorable David Obey 
Chairman 

The Honorable Jerry Lewis 
Ranking Minority Member 
Committee on Appropriations 
House of Representatives 

The Honorable Bennie Thompson 
Chairman 

Committee on Homeland Security 
House of Representatives 

The Honorable Barbara Boxer 
United States Senate 

The Honorable Dianne Feinstein 
United States Senate 

The Honorable Bob Filner 
House of Representatives 

The Honorable Doris Matsui 
House of Representatives 

The Honorable Mike Thompson 
House of Representatives 

The Honorable Susan Davis 
House of Representatives 
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Appendix I: Briefing Slides 
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Objectives 



In response to a legislative mandate, congressional request, 
and based on discussions with relevant congressional staff, 
we addressed the following: 

• What process did DHS use to allocate UASI grants in 
fiscal year 2006? 

• How did DHS determine the urban areas that were 
eligible to apply for UASI grants in fiscal year 2006? 

• How did DHS determine award amounts in fiscal year 
2006? 

• What changes, if any, does DHS plan to implement in 
fiscal year 2007 to its UASI award determination 
process? 



2 



GAO-07-381R Urban Area Security Initiative 



A GAP 

T«B^— Accountability * Integrity • Reliab 

Scope and Methodology 



We analyzed DHS documents and interviewed DHS officials 
about: 

• The UASI grant determination process in FY06 including 

• How risk analysis is used to inform decisions 

• How the peer review process is conducted 

• How allocation decisions are made 

• Planned changes to the FY07 grant determination 
process -- DHS Correspondence of October 25, 2006. 

We did our work between September 2006 and November 
2006, in accordance with generally accepted government 
accounting standards (GAGAS). 
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Results in Brief: 



In FY06, DHS combined a risk analysis and effectiveness assessment - using 
empirical analytical methods and policy judgments-to select eligible urban 
areas and allocate UASI funds. 

• DHS estimated the relative risk of a successful terrorist attack to urban 
areas, identified candidate urban areas, and from these selected 35 
urban areas that were eligible to apply for FY06 grants. 

• DHS introduced an effectiveness assessment-a peer review 
process — to assess and score the effectiveness of the proposed 
investments submitted by the eligible applicants. 

• DHS based awards on a decision to allocate a larger proportion of 
funds to areas considered to be both at highest risk and most effective. 

For FY07, DHS plans to retain the general process used in FY06 but key 
decisions have yet to be determined. 
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Overview of the grant determination process 



Risk analysis model 



Risk estimator 
R=TxVxC 



Relative risk 
estimate 



Relative risk 



riigible 

urban 



Cut 

point 




1 10 20 30 40 50 60 70 80 90 
Urban area 



Effectiveness analysis 



Peer review 
of investment 
justifications 



Allocation process 



Effectiveness/risk matrix 






PO 








n 


o 



Effectiveness score ->■ 

O Circles are sized relative 
to allocation amount 



Source: GAO analysis of DHS documents and information provided in interviews. 



GAO-07-381R Urban Area Security Initiative 



A GAP 

™a^— Accountability * Integrity • Reliab 

Risk analysis model used in determining 
eligible areas 

• For FY06, DHS adopted a general approach to estimating 
Risk{= Threat x Vulnerability x Consequences), but 
calculated risk by combining the results from two 
complementary models: 

• Asset-based risk and 

• Geographic-based risk 
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Risk analysis model used in determining 
eligible areas 
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Data used to measure asset and geographic 
risk 

DHS collected quantitative data such as: 

• Number of each type of asset 

• Number of intelligence reports and other indicators of 
geographic Threat 

• Population and related measures for geographic 
Consequences 

DHS generated data such as: 

• Threat values assigned to assets based on current and 
credible intelligence 

• Vulnerability values for assets derived by mapping attack 
scenarios against asset types 
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Judgments involved in estimating risk 



Judgments 


Outcomes in FY06 


Identifying critical assets. 


DHS identified 38 asset types. 


Selecting appropriate geographic urban 
area boundaries, known as a footprint, for 
risk estimation purposes. 


DHS identified the footprint by (1) applying 
population or threat criteria to cities, (2) 
merging contiguous cities, then (3) 
drawing a 10-mile buffer area around the 
merged area. 


Determining candidates that are 
considered for UASI grant eligibility. 


90 candidate Urban Areas were identified. 


Determining what data are integrated and 
how reliability is assessed. 


62 data variables from public and private 
sources, and DHS-generated data, were 
used. DHS generally does not verify data; 
outside sources were assumed to be 
reliable. 


Combining asset and geographic risk 
scores after weighting. 


Asset and geographic risk were weighted 
as 0.5 and 1.0 respectively, then summed. 

Total risk = (0.5)Asset + (I.O)Geographic 
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Judgments in eligibility decisions in FY06 



• DHS computed risk scores for the 90 candidate areas and 
plotted them on a relative risk curve (see next slide). 

• DHS determined the cut point on the relative risk curve that 
would determine which Urban Areas were eligible to apply for 
grants, according to DHS officials. 

• The cut point the DHS Secretary selected resulted in 35 
Urban Areas, which accounted for 85 percent of total 
estimated risk. 

• DHS decided to allow an additional 1 1 to apply. These Urban 
Areas received UASI funds in FY05, but were outside the cut 
point in FY06. 
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Risk estimates used to inform eligibility 
decisions 
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Effectiveness assessment added in FY06 



• DHS assessed the applications submitted by the 46 eligible 
urban areas. 

• DHS used a peer-review process to assess and score the 
effectiveness of proposed investments by: 

• engaging the states in identifying and selecting peer 
reviewers, 

• having peer reviewers individually score investments, and 

• assigning peer reviewers to panels to make final 
effectiveness score determinations. 

• According to DHS, this assessment encouraged urban areas 
to identify their needs and develop specific initiatives to 
address these needs. 
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Allocation process based on both risk and 
effectiveness scores 

• In fiscal year 2006, DHS combined risk (2/3 of total) and 
effectiveness scores (1/3) to determine amounts to allocate. 

• DHS used a 2 x 2 matrix to make decisions and set priorities 
(see next slide). 

• DHS created four quadrants which were given different 
funding priorities; each quadrant had its own funding 
allocation. 

• DHS plotted each eligible urban area's risk and 
effectiveness score on the matrix. 

• Some policy decisions were involved in determining how 
much funding applicants in each quadrant received. 
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A GAP 

O**^— Accountability * Integrity * Reliability 



DHS used a 2 x 2 matrix to make decisions 
and set priorities 




Circles are sized relative 
to allocation amount 

I. Higher risk/higher effectiveness 

II. Higher risk/lower effectiveness 

III. Lower risk/higher effectiveness 

IV. Lower risk/lower effectiveness 

Source: GAO analysis ot DHS documenls and informal ion provided in interviews^ 
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A GAP 

™a^— Accountability * Integrity • Reliab 

Plans for FY 2007 grant process 



• DHS planning to retain the structure and approach used in 
FY 2006 

• assessing risk to determine eligibility, 

• scoring effectiveness of proposed investments, and 

• combining risk and effectiveness to determine UASI 
award amounts. 

• Planned changes include: 

• DHS Office of Intelligence & Analysis will apply judgments 
to threat information. 

• Key eligibility and allocation decisions are still under 
consideration, according to DHS officials. 
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A GAP 

O**^— Accountability * Integrity * Reliability 



Concluding observations 



• Inherent uncertainty associated with estimating risk of 
terrorist attack requires policy and analytic judgments. 

• DHS performed some analyses to assess the sensitivity of its 
risk estimates to alternative policy and analytical judgments. 

• DHS has adopted a process of "continuous improvement" to 
its methods for estimating risk and measuring applicants' 
effectiveness. 
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Appendix II: DHS's Approach to Risk Analysis for Fiscal Year 2006 



In fiscal year 2006, DHS made enhancements to its approach to estimating risk that involved 
incorporating stakeholder feedback and three risk factors — threat, vulnerability, and consequence. 
Other models and methodologies of assessing risk also include these three risk factors. However, the 
inherent uncertainties associated with estimating the risk of terrorist attacks required DHS to make 
numerous policy and analytic judgments. The results of DHS's risk assessment were used to inform two 
key grant decisions in fiscal year 2006: (1) eligibility of urban areas to apply for UASI funding and 
(2) funding amounts. 

DHS has developed a flexible approach to assessing risk of terrorist attacks that considers several 
factors, including stakeholder feedback. In developing DHS's fiscal year 2006 UASI grant determination 
process, DHS considered agency goals and statutory responsibilities related to risk management. DHS's 
fiscal year 2006 funding criteria — based on relative risk and effectiveness of proposed solutions to 
identified needs — align federal resources with the national priorities established by the Interim National 
Preparedness Goal. In addition, DHS solicited feedback from states, territories, and local governments to 
increase transparency and held discussions with stakeholders and experts such as the RAND 
Corporation regarding data analysis. 7 For example, in May 2005, DHS hosted a meeting to solicit 
feedback on the fiscal year 2005 risk formula, which was attended by representatives from 12 states or 
urban areas and from law enforcement and fire service associations. DHS officials reported that changes 
to the fiscal year 2006 risk estimation model for fiscal year 2007 were based on feedback received, given 
the data were relevant and the changes could be applied to all urban areas during the data collection 
phase. However, agency officials said that implementing these suggestions varied in cost and time from 
minimal to very costly and time-consuming. Additionally, we were told that incorporating suggestions 
from states, territories, and local areas may not add significant value to outcomes, but DHS did not test 
the impacts of these changes. Where possible, DHS has integrated approaches with the intent of 
improving the model and its approach to estimating risk. 

DHS changed its definition of risk in fiscal year 2006 to incorporate common components from other 
models. DHS defined risk by three principal components: threat, or the likelihood of a type of attack that 
might be attempted; vulnerability, or the likelihood of a successful attack with a particular attack 
method; and consequence, or the potential impact of a particular attack. Other risk assessment models 
also use these three components. For example, other federal agencies have adopted some form of threat, 
vulnerability, and consequence into their risk management frameworks. DOD's risk management 
approach includes threat and vulnerability assessments that identify potential threats and weaknesses 
that may be exploited by those threats. The Department of Justice provided guidance to law 
enforcement executives on how to assess risk of terrorism to an asset by combining assessments of 
threat, vulnerability, and criticality, which evaluates the likely impact if an identified asset is lost or 
harmed by specific events. Additionally, the RAND Corporation argues that threat, vulnerability, and 
consequences play a significant role in assessing risk to urban areas and defines risk in a way that links 
these three components. Further, in February 2005, the Congressional Research Service reported that 
many risk assessment models and methodologies consisted of identifying critical assets, evaluating 
threats, assessing the vulnerabilities of critical assets, and determining the expected consequences of 



The RAND Corporation is a nonprofit policy research and analysis institution that has conducted national security research for 
the U.S. Department of Defense, the intelligence community, and key allied governments and ministries of defense. In addition, 
RAND operates three federally funded research and development centers that focus on national security issues. 
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specific types of attack on specific assets. For instance, the report noted the American Petroleum 
Institutes and the National Petrochemical and Refiners Association defined risk as a function of 
consequences of a successful attack against an asset, and likelihood of a successful attack against an 
asset, where likelihood is defined as the attractiveness of the target to the adversary based on the 
adversary's intent and the target's perceived value to the adversary, degree of threat based on 
capabilities, and degree of vulnerability of the asset. 

In fiscal year 2006, DHS combined two risk assessments — asset-based risk and geographic-based risk — 
that were both based on threat, vulnerability, and consequence to determine the relative risk of a 
successful terrorist attack to urban areas. The asset-based risk assessment analyzed the intent and 
capabilities of an adversary to successfully attack any of 38 asset types, such as a chemical, plant, dam, 
or commercial airport, using one of 14 different attack scenarios (e.g., nuclear explosion or vehicle- 
borne improvised explosive device.) DHS identified the list of 38 asset types, and according to DHS 
officials, it collected over 200,000 individual assets obtained from public and private sector sources. 
Geographic risk considered the general geographic characteristics of an area mostly independent of the 
area's assets using counts of information, such as suspicious incident reports, FBI cases, and population. 
Table 1 describes what we know about how each component of asset and geographic risk were 
calculated. According to DHS, the two estimates of risk, asset-based and geographic, were 
complementary providing a micro- and macro-prospective of risk, respectively. Furthermore, while 
DSH's risk analysis was largely based on population and population density in previous years, a DHS 
official told us that legislative language directed DHS to look at threats to infrastructure, which was 
partly why DHS added the asset-based analysis. 



"Congressional Research Service, Risk Management and Critical Infrastructure Protection: Assessing, Integrating, and 
Managing Threats, Vulnerability, and Consequences, (Washington, D.C.: February 2005). 



Page 27 



GAO-07-381R Urban Area Security Initiative 



Table 1 : Description of Asset-Based and Geographic Risk Computations in Fiscal Year 2006 

Component Description 
Asset-based risk 

Threat DHS used information from the intelligence community, such as communications intercepts and 

assessments of the abilities of adversaries to carry out various types of attacks. This information 
was evaluated on two main criteria, the intent and capability of the group making the threat. The 
strategic intent of an adversary is based on the "chatter factor" and "attractiveness," which is partly 
determined by how closely the results of a type of attack align with high-level objectives of an 
adversary. We learned that information used in this component for fiscal year 2006 was based on 
the terrorist group viewed as having the "greatest capabilities" across all attack scenarios. How the 
variables were calculated to get a measurement of threat to a particular asset type was not 
specified. 

Vulnerability To measure the vulnerability of an asset type, DHS used internal subject matter experts who 

analyzed the general attributes of an asset type against various terrorist attack scenarios. These 
subject matter experts conducted site vulnerability analyses on a sample of sites from the asset 
type to catalog attributes for the generic asset. Experts evaluated vulnerability by attack scenario 
and asset type pairs (e.g., nuclear explosion against a chemical plant) and assigned an ordinal 
relative value (1 , 2, or 3) to the pair based on 10 major criteria (e.g. electronic detection, access 
control, etc.). 

Consequences The mode of attack associated with the greatest likelihood of success was used to assess the 

consequence that would result from such an attack on the asset type. DHS used four categories of 
consequences— human health, economic, strategic mission, and psychological— for this 
assessment, which were identified in the National Strategy for Infrastructure Protection. These 
categories were weighted and then summed. Details about what data were used to calculate or 
simulate consequence were not specified. 

Geographic risk 

Threat To measure threat to a geographic area, DHS used counts of data from seven variables— total of 

intelligence community reports, total of FBI investigations, total of DHS/lmmigration and Customs 
Enforcement (ICE) investigations, total of suspicious incidents, total of ICE I-94 information for 
specific countries, total of international visitors from specific countries, and total of vessels from 
specific countries. Weights were assigned to each variable, then summed. 

Vulnerability DHS used total of international visitors and miles of designated Waste Isolation Pilot Plant (WIPP) 

route to assess the vulnerability of a geographic area. Details on how the two were computed to 
achieve a measure of vulnerability for a given area were not specified. 



Consequences DHS used three of the four categories used to assess consequences to asset types to assess the 

consequences to a geographic area. DHS did not factor economic consequences to urban areas in 
fiscal year 2006. According to a DHS official, it did not have a UASI-specific economic measure in 
fiscal year 2006, but has added it to the model for fiscal year 2007 using gross metropolitan product 
data. In fiscal year 2006, DHS used various population types, population density, total of defense 
industrial base facilities, total of military installations, and total of large gatherings/special events to 
measure consequence. A description of how the variables were computed to achieve a measure of 
consequence for a given area was not specified. 

Source: GAO analysis of DHS data. 

In calculating asset-based and geographic risk, DHS made a number of policy and analytic judgments 
because of uncertainties in estimating risk of terrorism. There are inherent uncertainties associated with 
estimating the risk of a terrorist attack, due to various factors such as limited information on actual 
attacks; limited information on goals, capabilities, and adaptability of terrorist groups; and differences in 
views about how to combine data about threat, vulnerability, and consequences into a risk methodology. 
Given uncertainties, policy and analytic judgments are required to inform the estimation process. For 
example, there are a number of judgments involved in estimating asset-based and geographic risk scores 
with various implications and limitations. Table 2 describes some of the judgments DHS made in 
estimating risk. 
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Table 2: Judgments Used in Estimating Asset-Based and Geographic Risk in Fiscal Year 2006 and Potential 
Implications and Limitations 

Judgment Potential Implications and Limitations 



Asset-based risk 



Identifying critical assets — 38 asset 
types for fiscal year 2006. 



DHS assessed risk scores for generic types of assets, such as bridges. 
Alternatively, different risk models assess the threat and vulnerability of a 
specific asset, such as the Golden Gate Bridge, and factor in consequences 
from an attack to that specific asset. While determining which assets are critical 
can be a subjective judgment, there may also be a wide variance regarding the 
criticality of assets within a particular asset type. 



Determining threat to assets from 
intelligence data on chatter, 
attractiveness of assets as targets, and 
strategic intent and capabilities of 
terrorist groups to attack assets. 



The capabilities of various terrorist groups are constantly changing, and 
there is no known method for predicting future motivations of adversaries. 



Estimating vulnerability of assets to 
attack from internal subject matter 
experts who assigned values using 
various attack scenarios — pairing of 
each of the 38 asset types to 1 4 attack 
scenarios for fiscal year 2006 



DHS noted the limitation of this approach in determining vulnerability for 
generic asset types and would have liked to have conducted site visits for 
all assets instead of a sample for each asset type. Details about what 
information internal subject matter experts used to assign a value for 
vulnerability was not specified. DHS officials told us that using an ordinal 
value (1 , 2, or 3) to measure vulnerability did not allow DHS to assess the 
differences in magnitude between the asset-scenario pairs with different 
values. Therefore, DHS used cardinal values (0-100) for fiscal year 2007 
analyses. 



Determining consequences of attack in 
terms of human health, economic, 
strategic mission, and psychological 
and assigning weights to each 
component. 



This method does not account for multiple or simultaneous attacks on 
assets because of lack of data and DHS' inability to compute these 
scenarios. DHS officials stated the current model does not address 
simultaneous multiple attacks. 

DHS officials reported many challenges to modeling consequences from 
multiple or simultaneous attacks on assets including answering modeling 
questions, such as who should determine what combination of location or 
mode of attacks was most likely (e.g. vehicle-borne improvised explosive 
device at a mall, plus a suicide bomber at a federal building). Additionally, 
according to agency officials, even if DHS were able to select the most 
likely multi-attack, it is very difficult to estimate the potential 
interdependencies and consequences. DHS continues to devise a way to 
integrate these issues into its risk model. DHS acknowledges the 
uncertainty of consequence values used in the model, but does not know 
of available databases for consequence information for all asset-scenario 
pairs. However when data are available, DHS uses them, such as with its 
use of EPA's database of "worst-case" scenarios from chemical releases. 



DHS did conduct sensitivity analysis for the consequence weights and 
told us that risk results did not change much under different assumptions 
about weights and that this may have been due to the fact that the 
consequences were positively correlated. 
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Geographic risk 



Geographic threat was based on 
information from the intelligence 
community, such as reports, and 
numbers of FBI investigations, ICE 
investigations, and ICE 1-94 data. 

Vulnerability was assessed in relation 
to total international visitors and miles 
of designated WIPP routes. 

Three types of consequences were 
assessed — human health, strategic 
mission, and psychological— and data 
on population and other factors were 
used. 

Source: GAO analysis of DHS data. 

The results of the asset-based and geographic risk calculations were combined to determine a total risk 
score for a candidate area. Combining these scores involved (1) determining the values of parameters; 
(2) normalizing the values; (3) weighting factors, 0.5 for asset and 1.0 for geographic; and (4) summing 
the values. Before adding the two estimates of risk, DHS made a judgment to weight geographic risk 
twice as much as asset-based risk since the potential loss of lives within an area was factored into how 
geographic risk was calculated, according to DHS officials. In determining the appropriate weights, DHS 
reported that it conducted limited sensitivity analysis for the weights applied to the asset and geographic 
risk scores, but that it would have been a useful tool to help inform decision makers about eligible 
candidate areas. During our review, we conducted sensitivity analysis for the weights assigned to asset- 
based and geographic risk estimates, which took an analyst a few hours to complete. By varying the 
weights, DHS could identify a subset of candidate areas that fall in and out of the cutoff point for UASI 
grant eligibility, which could justify the decision — 35 urban areas. DHS has approached the National 
Infrastructure Simulation and Analysis Center to conduct work to identify sources of uncertainty, which 
could help better inform analytic judgments. 9 

DHS used essentially the same risk assessment methods for fiscal year 2007. According to DHS officials, 
the most significant change to the model is in how threat was assessed. In fiscal year 2006, DHS used 
counts of data from the intelligence community to estimate threat to asset types and geographic areas. In 
fiscal year 2007, DHS's Homeland Infrastructure Threat and Risk Analysis Center — a joint unit of the 
Office of Intelligence and Analysis and the Office of Infrastructure Protection — will assess current and 
trend threat data to assign a single threat value for each asset type and geographic area using a tiered 
approach. Other changes were in response to stakeholder feedback. For example, DHS expanded the 
number of asset types in its assessment from 38 in fiscal year 2006 to 47 in fiscal year 2007, based on the 
feedback provided to DHS from users, such as states and local governments. 

Identifying Eligible Urban Areas 

In applying its risk assessment to determine the urban areas that were eligible to receive UASI grants, 
DHS first had to determine the geographic boundaries or footprint of candidate urban areas within 
which data were collected to estimate risk. Table 3 identifies the footprints for eligible urban areas in 
fiscal year 2006. It used data from various sources to calculate risk scores; the sources included 
information from federal agencies; proprietary data on assets; and intelligence data on threats, 
suspicious incidents, and other indicators of threats. Appendix III further describes the data sources 
used by DHS to assess risk. 



9 The National Infrastructure Simulation and Analysis Center is a virtual center that includes national laboratories, such as 
Sandia, Los Alamos, and Argonne National Laboratories. 



In general, we know very little about how DHS estimated geographic risk 
and judgments regarding what parameters are used in assessing threat, 
vulnerability, and consequence were not specified. RAND has indicated 
limitations in using simple indicators such as counts of data to assess 
risk, although there is no theoretical and empirical basis for deciding what 
counts should be included and in what proportion. 
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On the basis of comments from state and local governments, DHS chose to redefine the footprint for 
fiscal year 2006. DHS took several steps to identify this footprint; these included: 

■ Identifying areas with population greater than 100,000 persons and areas (cities) that had any 
reported threat data during that past year. For fiscal year 2006, DHS started with a total of 266 cities. 

■ Combining cities or adjacent urban counties with shared boundaries to form single jurisdictions. For 
fiscal year 2006, this resulted in 172 urban areas. 

■ Drawing a buffer zone around identified areas. A 10-mile buffer was then drawn from the border of 
that city/combined entity to establish candidate urban areas. 10 This area was used to determine what 
information was used in the risk analysis, and represents the minimum area that had to be part of the 
state/urban areas defined grant application areas. 

In fiscal year 2005, the footprint was limited to city boundaries (and did not include the 10-mile buffer 
zone). According to DHS, for fiscal year 2006, it considered other alternatives such as a radius from a 
city center, although such a solution created apparent inequities among urban areas. DHS incorporated 
buffer zones at the suggestion of stakeholders, although this action resulted in making the analysis more 
difficult, according to a DHS official. In addition, DHS officials told us the steps taken to determine the 
footprint were based on the "best fit," as compared with other alternatives. DHS did not provide details 
on what criteria this comparison was based on. 



Table 3: Footprint of Urban Areas Eligible for UASI Grants in Fiscal Year 2006 


State 


Eligible urban area 


Geographic area captured in the data count 


Previously 
designated 
urban areas 
included 


AZ 


Phoenix Area 3 


Chandler, Gilbert, Glendale, Mesa, Peoria, Phoenix, 
Scottsdale, Tempe, and a 10-mile buffer extending from the 
border of the combined area. 


Phoenix, AZ 


CA 


Anaheim/Santa Ana Area 


Anaheim, Costa Mesa, Garden Grove, Fullerton, Huntington 
Beach, Irvine, Orange, Santa Ana, and a 10-mile buffer 
extending from the border of the combined area. 


Anaheim, CA; 
Santa Ana, CA 




Bay Area 


Berkeley, Daly City, Fremont, Hayward, Oakland, Palo Alto, 
Richmond, San Francisco, San Jose, Santa Clara, 
Sunnyvale, Vallejo, and a 10-mile buffer extending from the 
border of the combined area. 


San Francisco, 
CA; San Jose, 
CA; Oakland, CA 




Los Angeles/Long Beach Area 


Burbank, Glendale, Inglewood, Long Beach, Los Angeles, 
Pasadena, Santa Monica, Santa Clarita, Torrance, Simi 
Valley, Thousand Oaks, and a 10-mile buffer extending from 
the border of the combined area. 


Los Angeles, 
CA; Long Beach, 
CA 




Sacramento Area 8 


Elk Grove, Sacramento, and a 10-mile buffer extending from 
the border of the combined area. 


Sacramento, CA 




San Diego Area" 


Chula Vista, Escondido, and San Diego, and a 10-mile 
buffer extending from the border of the combined area. 


San Diego, CA 


CO 


Denver Area 


Arvada, Aurora, Denver, Lakewood, Westminster, Thornton, 
and a 10-mile buffer extending from the border of the 
combined area. 


Denver, CO 



10 Buffer zone extensions were considered for chemical plants (25 miles) and nuclear power plants (50 miles). According to DHS 
officials, these distances were selected based on plume effects influenced by research conducted by the Department of Energy. 
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State 


Eligible urban area 


Geographic area captured in the data count 


Previously 
designated 
urban areas 
included 


DC 


National Capital Region 


National Capital Region and a 10-mile buffer extending from 
the border of the combined area. 


National Capital 
Region, DC 


FL 


Fort Lauderdale Area 


Fort Lauderdale, Hollywood, Miami Gardens, Miramar, 
Pembroke Pines, and a 10-mile buffer extending from the 
border of the combined area. 


N/A 




Jacksonville Area 


Jacksonville and a 10-mile buffer extending from the city 
border. 


Jacksonville, FL 




Miami Area 


Hialeah, Miami, and a 10-mile buffer extending from the 
border of the combined area. 


Miami, FL 




Orlando Area 


Orlando and a 10-mile buffer extending from the city border. 


Orlando, FL 




Tampa Area 3 


Clearwater, St. Petersburg, Tampa, and a 10-mile buffer 
extending from the border of the combined area. 


Tampa, FL 


GA 


Atlanta Area 


Atlanta and a 10-mile buffer extending from the city border. 


Atlanta, GA 


HI 


Honolulu Area 


Honolulu and a 10-mile buffer extending from the city 
border. 


Honolulu, HI 


IL 


Chicago Area 


Chicago and a 10-mile buffer extending from the city border. 


Chicago, IL 


IN 


Indiananoli^ Area 

1 1 1 V-l 1 CI 1 1 CI k_/ \A 1 1 1 V 1 OCA 


Indiananoli^ and a 10-mile huffer extendinn from the ritv 

1 1 1 CI 1 LA. 1 1 CI C/ \J 1 1 C4. 1 1 V-l CI 1 \J 1 1 1 1 1 C/ UU 1 1 w 1 C/ /\ 1 1 \A 1 1 1 VJ II C/l II L 1 1 \j Kj 1 L y 

border. 


Indiananolis IN 

1 1 1 \-A 1 CJ. 1 1 C4. kv V_/ 1 IOj 1 1 >l 


KY 


1 oijkvNIp Arpa a 

1 \J UIOV lilt; / Y 1 


1 oui^villp and a 10-milp huffpr pxtpndinn from thp ritv 
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border. 
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LA 


Baton Rouop Arpa a 
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New Orleans Area 


New Orleans and a 10-mile buffer extending from the city 
border. 


New Orleans LA 


MA 


Boston Area 


Boston, Cambridge, and a 10-mile buffer extending from the 
border of the combined area. 


Boston, MA 


MD 


Baltimore Area 


Baltimore and a 10-mile buffer extending from the city 
border. 


Baltimore, MD 


Ml 


Detroit Area 


Detroit, Sterling Heights, Warren, and a 10-mile buffer 
extending from the border of the combined area. 


Detroit, Ml 


MN 


Twin Cities Area 


Minneapolis, St. Paul, and a 10-mile buffer extending from 
the border of the combined entity. 


Minneapolis, 
MN; St. Paul, 
MN 


MO 


Kansas City Area 


Independence, Kansas City (MO), Kansas City (KS), 
Olathe, Overland Park, and a 10-mile buffer extending from 
the border of the combined area. 


Kansas City, MO 




St. Louis Area 


St. Louis and a 10-mile buffer extending from the city 
border. 


St. Louis, MO 


NC 


Charlotte Area 


Charlotte and a 10-mile buffer extending from the city 
border. 


Charlotte, NC 


NE 


Omaha Area 8 


Omaha and a 10-mile buffer extending from the city border. 


Omaha, NE 
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State 


Eligible urban area 


Geographic area captured in the data count 


Previously 
designated 
urban areas 
included 


NJ 


Jersey City/Newark Area 


Elizabeth, Jersey City, Newark, and a 10-mile buffer 
extending from the border of the combined area. 


Jersey City, NJ; 
Newark, NJ 


NV 


Las Vegas Area" 


Las Vegas, North Las Vegas, and a 10-mile buffer 
extending from the border of the combined entity. 


Las Vegas, NV 


NY 


Buffalo Area 8 


Buffalo and a 10-mile buffer extending from the city border. 


Buffalo, NY 




New York City Area 


New York City, Yonkers, and a 10-mile buffer extending 
from the border of the combined area. 


New York, NY 


OH 


Cincinnati Area 


Cincinnati and a 10-mile buffer extending from the city 
border. 


Cincinnati OH 




Cleveland Area 


Cleveland and a 10-mile buffer extending from the city 
border. 


Cleveland OH 




Columbus Arpa 


Columbus and a 10-mile buffer extending from the city 
border. 


Columbus, OH 




Toledo Area a 


Oregon, Toledo, and a 10-mile buffer extending from the 
border of the combined area. 


Toledo OH 


OK 


Oklahoma City Area 3 


Norman, Oklahoma City, and a 10-mile buffer extending 
from the border of the combined area. 


Oklahoma City, 
OK 


OR 


Portland Area 


Portland, Vancouver, and a 10-mile buffer extending from 
the border of the combined area. 


Portland, OR 


PA 


Philadelphia Area 


Philadelphia and a 10-mile buffer extending from the city 
border. 


Philadelphia, PA 




Pittsburgh Area 


Pittsburgh and a 10-mile buffer extending from the city 
border. 


Pittsburgh, PA 


TN 


Memphis Area 


Memphis and a 10-mile buffer extending from the city 
border. 


Memphis, TN 


TX 


Dallas/Fort Worth/Arlington 
Area 


Arlington, Carrollton, Dallas, Fort Worth, Garland, Grand 
Prairie, Irving, Mesquite, Piano, and a 10-mile buffer 
extending from the border of the combined area. 


Dallas, TX; Fort 
Worth, TX; 
Arlington, TX 




Houston Area 


Houston Pasadena and a 10-mile buffer extendinn from 

I iuuoiui I, I ciodi-ic. I id, diivj ci I \J Mine, uumci I I 1 1 I y iiui 1 1 

the border of the combined entity. 


Houston TX 




San Antonio Area 


San Antonio and a 10-mile buffer extending from the city 
border. 


San Antonio, TX 


WA 


Seattle Area 


Seattle, Bellevue, and a 10-mile buffer extending from the 
border of the combined area. 


Seattle, WA 


Wl 


Milwaukee Area 


Milwaukee and a 10-mile buffer extending from the city 
border. 


Milwaukee, Wl 



Source: DHS. 

"Sustainment area: an urban area that received UASI funding in fiscal year 2005, but was not deemed eligible to apply through 
the fiscal year 2006 risk assessment. However, DHS extended eligibility to these areas for one additional grant cycle. 



On the basis of the risk assessment and a policy decision, DHS determined which urban areas were 
eligible to apply for UASI grants in fiscal year 2006. DHS estimated risk for 172 urban areas, but in 
determining eligible urban areas, it limited its analysis of risk to 90 candidate areas, based on a 200,000 
population threshold, and/or reports of credible threats. DHS performed calculations of relative risk for 
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the 90 urban areas. DHS combined the two risk assessment scores for 90 candidate urban areas to get 
their total relative risk score. These relative risk scores were plotted in order, then a cutoff point was 
selected that determined the number of urban areas eligible to apply for grants in fiscal year 2006 and 
defined the nation's most at-risk areas(Also see appendix I, page 20). According to DHS officials, the 
Secretary of Homeland Security selected a cut point that resulted in 35 urban areas, which accounted for 
85 percent of total estimated risk. A senior DHS official also told us that decision makers may bring 
other sensitive information — outside the risk model — to the table, but exactly what that information was 
or what priority that information held over other DHS goals was unclear. Further, DHS also extended 
eligibility to 11 sustainment areas — urban areas that participated in the program in fiscal year 2005, but 
were not identified as eligible through the risk analysis process in fiscal year 2006. This policy decision 
was made in order to foster long-term planning for program participants across fiscal years. According 
to DHS, any urban area not identified as eligible through the risk analysis process for two consecutive 
years will not be eligible for continued funding under the UASI program, but will continue to be eligible 
to receive funding from other DHS programs. 

DHS officials did not know the extent to which, if at all, the change in the definition of the footprint area 
between fiscal years 2005 and 2006 influenced estimates of risk. According to DHS officials, it would be 
very difficult to pinpoint the source of changes in risk analysis outcomes in fiscal year 2006, since there 
were changes made to the urban area's footprint, the structure of the model, and the data inputs (e.g., 
new annual threat data for geographic risk). However, DHS officials believe that the change in footprint 
in 2006 was associated with changes in relative risk of many urban areas. For example, by defining the 
footprint to go beyond city limits additional information, such as a nuclear power plant outside a city 
boundary or suburban population, was captured in the fiscal year 2006 risk assessment, which was 
previously not accounted for in several urban areas. As a consequence of the change in the footprint, 
DHS officials concluded that the relative risk of New York City and the National Capital Region declined 
compared to those of other urban areas. DHS could not determine how much of the decline was due 
only to the change in the footprint versus other components of the risk methodology that changed. 
While, as of November 2006, DHS expected to use the same definition for an urban area footprint for 
fiscal year 2007, it has yet to determine how eligibility for UASI funding will be decided. 
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Appendix III: Data Sources Used in DHS's Fiscal Year 2006 Risk Analysis 



In assessing risk for the UASI grant determination process in fiscal year 2006, DHS applied 57 types of 
data variables from sources such as (1) federal agencies; (2) state, territory, and local stakeholders; (3) 
private proprietary data; as well as (4) data compiled by DHS. Some data variables were populated from 
a combination of sources. Data variables from DHS and other federal government data sources made up 
36 asset-based and geographic data variables. Private proprietary data sources comprised 22 asset-based 
and geographic variables, of which, 7 variables were constituted exclusively with data from private 
proprietary sources. (See table 4 for details.) DHS officials told us that the National Asset Database 
(NADB) was not a data source for risk analysis since the database is not populated with relevant 
attributes. Our review of the list and sources of variables for the risk methodology that DHS provided us 
also reveal that NADB did not appear among the sources. 

DHS considered all data obtained for the risk model from the sources identified as reliable for the 
purposes of estimating risk, although DHS did not systematically test the reliability of the data used. This 
includes the intelligence data which DHS officials acknowledged that they had accepted from source 
agencies. DHS considered these data to be valid and reliable in the sense that DHS believed they 
appropriately measure the risk constructs for which they are collected (i.e., the data have face validity). 
According to DHS officials, to identify any data-related problems such as the validity of data used and 
any duplicative values, DHS made over 100 analytical runs of the fiscal year 2006 risk assessment model. 
These analyses revealed errors created by using buffer zones, which resulted in some individual assets 
being attributed to more than one urban area. 

Most of the data used by DHS in fiscal year 2006 were timely and appeared reliable. Our review of the 
data sources contained in table 4 show most of the data sources to be less than 2 years old, and most 
sources of data were from 2005 or 2006. All data supplied by private proprietary sources were less than 2 
years old. Data from federal sources on some asset-based variables were from 2002 or 2001. 

We performed a limited test on the reliability of the data sources, given the time constraints in 
conducting this review. We attempted to determine, as a result of prior or ongoing GAO work, whether 
any reliability assessments have been conducted on any of the data sets DHS obtained from proprietary 
sources, and if so, what were the results. To comply with GAO policies, we review the reliability of data 
whenever our work uses sources of data other than GAO-generated data to analyze and make 
conclusions in our work. Of the 57 data variables, we identified five data sources that were used in past 
GAO work, and found one of the sources has been questioned by GAO analysts, although our past work 
was not directly related to the specific type of data provided to DHS. Specifically, the provider of DHS's 
data for transoceanic cable landings asset type did not meet GAO's reliability standards, as our past 
work found internal control problems such as no mechanisms in place for the providers of the data to 
perform verifications. 

For fiscal year 2007, DHS reported that it will apply 69 types of asset-based and geographic data 
variables from these sources. Of these 69 variables, 38 were populated exclusively with data from a 
single source, and 24 asset-based variables were refined in fiscal year 2007 by adding additional data 
sources. Also, as we discussed in appendix II, DHS's Office of Intelligence and Analysis performed threat 
reviews and provided the Office of Grants and Training with a single threat value for each urban area 
and asset type. This is in contrast to fiscal year 2006, when DHS used total counts of threats and 
suspicious incidents. Data supplied directly by state and local governments for fiscal year 2007 analyses 
were current as of August 2006, except where otherwise noted. 
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Table 4: DHS Sources of Data Used in UASI Risk Analysis Model 



Source Used in: 



Type of data 


DHS Other 
federal 


State/ 
local 


Private 


Fiscal 
year 
2006 


Fiscal 
year 
2007 




Critical assets (assessed for threat, vulnerability, and consequences) 


Aqueducts 


X 






- 


X 


Arenas 




X 




- 


X 


Casinos 






X 


- 


X 


Chemical Manufacturing Facilities 


X a 






X 


X 


City Road Bridges 


x b 


X 




X 


X 


Colleges and Universities 


X 


X 




x 


X 


Commercial Airports 


X 


X 




x 


X 


Commercial Overnight Shipping Facilities 




X 


x° 


x 


X 


Convention Centers 




X 


X 


x 


X 


Dams 


X x a 


X 




x 


x 


tiecinciiy oeneraTion raciinies 




X 


X 


x 


x 


tiecinciiy ouusiauons 




X 


X 


x 


x 


Enclosed Shopping Malls 




X 


X 


x 


x 


Federal Office Buildings 


X 






- 


x 


Ferry Terminals - Buildings 




X 


X 


x 


x 


Financial Facilities 


X 


X 




x 


x 


Hospitals 




X 


X 


x 


x 


Hotels 




x a 


x° 


- 


x 


Hotels/Casinos 






x a 


x 


- 


Levee 


X 






x 


x 


Liquefied Natural Gas Terminals 


x a 






x 


x 


Maritime Port Facilities 


x° 






x 


x 


Mass Transit Commuter Rail and Subway 
Stations 






X 


x 


x 


National Health Stockpile Sites 


X 


X 




x 


x 


National Monuments and Icons 


X 






x 


x 


Natural Gas Compressor Stations 




X 


x a 


x 


x 


Non Power Nuclear Reactors 


x° 






x 


x 


Nuclear Power Plants 


x° 


X 




x 


x 


Nuclear Research Labs 


x° 


X 




x 


x 


Offshore Petroleum Import Terminals 


x° 








x 


Petroleum Pumping Stations 






x a 


x 


x 


Petroleum Refineries 






x a 


x 


x 
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Source Used in: 



Type of data 


DHS 


Other 
federal 


State/ 
local 


Private 


Fiscal 
year 
2006 


Fiscal 
year 
2007 


Petroleum Storage Tank Farms 






X 


X 


X 


X 


Pharmaceutical Plants 


X 










X 


Postal Shipping Hubs 


x a 










X 


Potable Water Treatment Facilities 




wta 

X 


X 




X 


X 


Primary And Secondary Schools 






X 


X 


X 


X 


Railroad Bridges 




X 


X 




X 


X 


ncii 1 1 uciu raooci lyci olciliui 10 




V 

A 


v 

A 




V 
A 


v 

A 


Railrnarl Xiinnolc 

PldlllUdU 1 UlllltMo 






v 

A 


Y 
A 


v 

A 


v 
A 


Rnarl l^*nmmi itor Xi innplc 
nuau wuiiiiiiuLt;i I ui ii icio 






V 

A 


Y 
A 


v 

A 


v 
A 


ocvvayc i i can i icm i l i duiimeo 




A 








v 
A 


Q+aHii imc 

OLdUIUI 1 li> 






Y 
A 


Y 
A 


v 

A 


v 
A 


Tall Commercial Buildings 








x° 


x 


x 


Telecommunications Telephone Hotels 






X 


X 


x 


x 


Theme Parks 






X 


X 


x 


x 


Trans Oceanic Cable Landings 








x a 


x 


x 


Under Water Mass Commuter Tunnels 


X 










x 




Asset vulnerability 


Fiscal Year 2006— ordinal value (e.g.: 1 , 2, 3) 




X° 






x 




Fiscal Year 2007 — cardinal value (e.g.: - 
100) 




X° 








x 




Asset threat 


Fiscal Year 2006- 


Strategic Intent 


x c 








x 




Chatter 


x° 








x 




Attractiveness 


x° 








x 




Capabilities 


x c 








x 




Fiscal Year 2007- 


Threat to Infrastructure (value between 
and 1) e 


X 










x 




Geographic consequence parameters 


Human Health (Population and Population 
Density) 




X° 






x 


X 


Human Health (Commuter Population) 




X 






x 


X 


Human Health (Visitor Population) 




X 






x 


X 
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Source Used in: 



Type of data 


DHS 


Other 
federal 


State/ 
local 


Private 


Fiscal 
year 

iUUU 


Fiscal 
year 


Human Health (Military Population) 




X 








X 


Economic (Gross Metropolitan Product) 




X° 








X 


Defense Industrial Base 




X 






X 


X 


Military Bases (Counts) 




X 






X 


X 


Levees (Counts) 






X 






X 


State Capitals (Yes/No— UASI only) 








X a 




X 



Special Events X c XX 



Geographic vulnerability 



Miles of International Border/Coastline 




X 




X 


Total International Visitors 


X 




X 


X 


ICE form 1-94 Visitors from Standard 
Industrial Classification Code (through City) 


X 






X 


ICE form 1-94 Visitors from Standard 
Industrial Classification Code (destination 
City) 


X 






X 



Miles of WIPP Routes XXX 



Geographic threat 



Fiscal Year 2006- 


Intelligence Community Reports 


X° 


X 




FBI Counts 


X° 


X 




ICE Investigations 


x c 


X 




Suspicious Incidents 


x° 


X 




ICE form 1-94 Visitors from Standard 
Industrial Classification Code (through City) 


x c 


X 




ICE form 1-94 Visitors from Standard 
Industrial Classification Code (destination 
City) 


x c 


X 




Fiscal Year 2007- 


Threat to Geographic Area (value between 
and1) e 


X 




X 



Source: GAO analysis of DHS data. 



Note: DHS provided us information on the sources of data used in the risk model on November 8, 2006 and, at the time of our 
review, indicated that the list for fiscal year 2007 was subject to change. In addition, the fiscal year 2007 data used have a 
publication date of 2006, unless otherwise noted. 
a Data sources with a publication date of prior to 2006. 
"Data sources with a publication date of either 2001 or 2002. 
TJata sources with a publication date not specified. 

"Denotes that publication dates for the variable were not provided for all sources. 

e DHS considered the variables used in the fiscal year 2006 risk model to assign a threat value between and 1 for fiscal year 
2007. 
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Appendix IV: DHS's Approach to Assessing Effectiveness for Fiscal Year 2006 



Fiscal year 2006 marked the first time that eligible urban areas completed and submitted an investment 
justification to formally request UASI funding, which DHS used to assess the anticipated effectiveness of 
the risk mitigation investments urban areas proposed. The investment justification included up to 
15 "investments" or proposed solutions to address homeland security needs identified by the states and 
urban areas through their strategic planning process. DHS and the states collaborated to identify and 
select peer reviewers who evaluated, discussed, and scored the investment justifications submitted by 
the 46 eligible urban areas. Reviewers on each of the 17 panels assigned scores for six investment 
justifications, which according to DHS officials were averaged to determine a final effectiveness score 
for each urban area. 

Purpose and Goals of the Effectiveness Assessment 

Given the uncertainties in estimating terrorism risk, DHS introduced the effectiveness assessment as an 
additional tool to inform DHS leaders when making allocation decisions. Specifically, the investment 
justifications allowed DHS to consider how the eligible urban areas planned to spend the grant money. 
While one identified goal of the UASI program is to address the needs of high-threat, high-density urban 
areas, DHS officials determined that it would be more useful for urban areas to suggest solutions for 
how to meet their self-identified needs within the investment justifications. In addition, DHS officials 
told us the emphasis on effectiveness was meant to avoid the potential bias that could have occurred 
from self-reported needs. The Interim National Preparedness Goal, which DHS described as a common 
planning framework to better understand preparedness levels, shape priorities, and focus expenditures, 
was in place for the first time for the fiscal year 2006 grant cycle. DHS reported that designing funding 
criteria that incorporated both risk and effectiveness was done to ensure that urban areas' expenditures 
were in alignment with the national priorities established by the Interim National Preparedness Goal. In 
particular, for the new effectiveness assessment the states and urban areas requested fiscal year 2006 
HSGP funding by submitting applications in support of their Homeland Security Strategies and related 
program planning documents. 11 In addition, according to DHS the new effectiveness assessment added a 
degree of competition to the grant determination process, which was a change from fiscal year 2005, 
when urban areas did not have to justify their planned use of the grant before they received the funding. 12 

Instead of DHS determining the effectiveness of the urban areas' applications, it decided to use peer 
reviewers, who were homeland security professionals and managers from various fields, to make this 
assessment. DHS reported that involving subject matter experts from federal, state, and local 
government agencies was done to ensure a fair and equitable peer review process. To learn best 
practices for distributing competitive grants and conducting peer reviews, DHS met with officials who 
run other competitive grant programs (Assistance to Firefighter Grants, Transit Security Grant Program, 
and the National Science Foundation). According to DHS, this approach to evaluating anticipated 
effectiveness seeks to recognize applicants for proposing relevant, innovative, and reasonable 
investments that will directly affect our nation's preparedness. 



"States and UASI areas were required to maintain a Homeland Security Strategy, which was meant to (a) provide a blueprint for 
comprehensive, enterprisewide planning for homeland security efforts and (b) provide a strategic plan for the use of related 
federal, state, local and private resources within the state or urban area before, during, and after threatened or actual domestic 
terrorist attacks, major disasters, and other emergencies. 

12 

In fiscal year 2006, as in fiscal year 2005, the states were required to notify DHS how they spent the funds. Within 60 days of 
the grant award, state administrative agencies were required to submit a prioritization of investments based upon the final grant 
award amounts and a certification that funds had been passed through to local units of government. 
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Preparing the Investment Justification 

DHS assessed effectiveness only for the applications submitted by the 46 eligible urban areas. Aside 
from the 11 sustainment areas, DHS stated that it did not allow areas that fell below the risk cut point to 
apply for a UASI grant because it did not want to set false expectations and create excessive work for 
candidate areas that were not going to receive funding. DHS provided states and urban areas with 
guidance that included instructions on completing the investment justification, the criteria peer 
reviewers would use to score the investment justifications, and an overview of how risk and 
effectiveness scores would be used to determine UASI allocations. DHS allowed each urban area to 
propose up to 15 investments, and for each investment, applicants were required to answer a total of 
17 detailed questions across four sections: background, regionalization, impact, and funding and 
implementation plan. DHS instructed urban areas to build investments that supported their state's 
Enhancement Plan, a program management plan to help states identify strengths and weakness within 
their homeland security programs and capabilities. 13 DHS cited this guidance as an example of how it 
encouraged states and urban areas to utilize the results of strategic planning efforts. DHS reported it was 
still determining how it would use the risk and effectiveness scores when allocating the UASI grants, and 
had not yet determined what weights would be applied to the risk and effectiveness scores at the time 
urban areas were completing their applications. Therefore, applicants did not know how much the 
effectiveness assessment would influence their grant amount. At the time of our review, DHS had not 
announced whether or not applicants will be provided with this information prior to submitting their 
fiscal year 2007 applications. In addition, in fiscal year 2006, applicants did not have access to the 
outcomes of the risk analysis or to specific threats to assets or their area for consideration when 
preparing the investment justifications. 

Forming the Peer Review Panels 

DHS engaged the states and territories in identifying and selecting the peer reviewers that would 
evaluate the investment justifications for their anticipated effectiveness. DHS provided some guidelines 
on what state officials should consider when nominating peer reviewers, and requested information, 
such as professional experience, about those nominated. DHS compiled a list of eligible peer reviewers 
from nominations made by the states and territories, and made its recommendations to the states based 
on the following high-level criteria: 

• the extent of the nominees' familiarity across multiple homeland security disciplines and their 
length of tenure, 

• the nominees' demonstrated experience managing an integrated homeland security program or 
initiative, 

• the nominees' familiarity with the HSGP (which was considered a benefit, but not required), and 

• whether or not the nominees represented the State Administrative Agency (if so, these nominees 
were prioritized). 

DHS allowed the State Administrative Agencies to make the final selection of peer reviewers from their 
state or territory, who included homeland security professionals and managers from a variety of 
disciplines, such as officials from law enforcement, fire service, emergency management, state 
homeland security, and public health. DHS arranged 17 panels to include one facilitator, one note-taker, 
and up to seven peer reviewers, representing states or territories, urban areas, and federal agencies. DHS 
reported arranging the panels to ensure a diverse mix of backgrounds and experience, and to avoid 
potential conflicts of interest by: 

• including representatives from the eastern, western, and central geographic regions, and from 
large- and small-population states; 



"According to DHS, in 2005 states conducted a Program and Capability Review and from this created an Enhancement Plan, 
which is meant to prioritize focus areas and develop high-level initiatives to address the most critical needs. In addition, the 
Enhancement Plan is the foundation for building an investment justification to request fiscal year 2006 HSGP funding. 
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• preventing reviewers from scoring their own state, territory, or urban area; and 

• avoiding reviewers scoring neighboring states, territories, or urban areas, where possible 
Overall, in DHS's view, the peer review panel process mitigated potential bias by requiring panelists to 
engage in discussion, justify their scores, and consider multiple perspectives. 

Reviewing and Scoring the Investment Justifications 

When scoring an urban area's investment justification, peer reviewers conducted an individual 
assessment of the applications, and subsequently discussed scoring in peer review panels during a week- 
long conference. Each peer reviewer was responsible for reviewing six investment justifications, which 
included roughly 60 investments, on an individual basis over the course of 2 V2 weeks, and then 
submitted their scores, along with explanatory comments, to DHS. Specific scoring criteria were 
developed by DHS for the peer reviewers to use and were provided to states and urban areas about a 
month prior to the March 2, 2006, HSGP application deadline. To score each individual investment, the 
reviewers evaluated the responses to the 17 questions, comparing them to detailed criteria and the 
state's Enhancement Plan to ensure the proposed investments were in alignment. Peer reviewers also 
scored the overall submission, so DHS provided the peer reviewers with criteria to consider the 
investment justification as a whole. By scoring the investment justification as a whole, DHS sought to 
reward innovative, forward-leaning approaches. The scoring criteria are summarized in table 5. 
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Table 5: Factors Peer Reviewers Considered When Scoring Investment Justifications in Fiscal Year 2006 



Individual Investment 
Section description 

Background : 

Applicants were asked to summarize the investment, its 
purpose, and how it will support the Enhancement Plan, 
state/urban area homeland security strategies, and 
national priorities and target capabilities. 

Includes four questions with multiple criteria for each 
question (a total of 14 criteria for the section). 

Reqionalization : 

Applicants were asked to describe the investment's 
demographic and geographic area, and the urban area's 
plans for regional collaboration, stakeholder engagement, 
and an implementation approach to support the 
investment. 

Includes three questions with multiple criteria for each 
question (a total of 14 criteria for the section). 



Examples 

Question: Provide a summary description of this Investment 
and its purpose. 

Criteria: 

• Articulates clear end result of using fiscal year 2006 HSGP 
funds; 

• Explains how outcomes relate to the purpose. 



Question: Explain how the state/urban area is organizing to 
implement this Investment over the identified geographic 
areas(s). 
Criteria: 

• Discusses regional partnerships; 

• Discusses mitigating duplication of effort. 



Impact: 

Applicants were asked to describe anticipated impacts of 
the investment, how requested funds will help achieve the 
impacts, how the investment will decrease or mitigate 
risk, and what the potential risks of not funding the 
investment would be. 



Question: Discuss how the implementation of this Investment 

will decrease or mitigate risk. 

Criteria: 

• Targets specific consequences, vulnerabilities, and threats; 

• Provides a rationale of choices. 



Includes three questions with multiple criteria for each 
question (a total of 1 1 criteria for the section). 



Funding and implementation plan : 
Applicants were asked to describe the investment's 
funding plan; describe the planned implementation and 
oversight approach of the management team, provide an 
implementation timeline with milestones, identify potential 
challenges to effective implementation and how they will 
be addressed and mitigated, and describe the planned 
duration and long-term sustainability plans of the 
investment after fiscal year HSGP funds are expended. 



Question: Identify potential challenges to the effective 
implementation of this Investment (e.g., stakeholder buy-in, 
sustainability, aggressive timelines). 
Criteria: 

• Describes necessary steps required for successful 
implementation and describes potential challenges; 

• Explains why the identified implementation challenges are 
challenges to this Investment. 



Includes seven questions with multiple criteria for each 
question (a total of 27 criteria for the section). 



Overall investment justification submission 

Criteria: 

• Overall relevance to implementation of the Interim National Preparedness Goal; 

• Connection to both the spirit and scope of the Enhancement Plan; 

• Extent to which the individual investments relate to each other to portray a complete picture of plans for the homeland 
security program; 

• Innovativeness of the proposed solutions to address needs; 

• Overall feasibility and reasonableness of proposed solutions. 



Source: GAO analysis of DHS documents. 
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After peer reviewers submitted preliminary scores based on their individual review, DHS identified the 
questions that received the greatest range of scores. Then the reviewers participated in a week-long 
conference, where the panels of peer reviewers discussed and scored each individual investment and the 
investment justification submission as a whole. Each panel had a facilitator, whose role according to 
DHS was to focus the discussions on those questions that received the greatest range of scores, ensure 
that the scoring criteria were consistently applied, and to help the panel develop feedback for the states, 
territories, and urban areas. In addition, subject matter experts were on call to answer questions that 
arose. During the conference, peer reviewers could revise their initial scores if desired, based on panel 
discussions. DHS computed the final scoring of the investments and the whole investment justification 
submission and then combined them to determine an overall effectiveness score. Specifically, peer 
reviewers provided scores for each of the investments in their assigned investment justifications based 
on evaluation criteria, and DHS told us it averaged the reviewers' scores together for each urban area. 14 
DHS reported it selected the median score as the final total "investment score" for each urban area. In 
addition, according to DHS, peer reviewers provided a score for each overall investment justification 
submission they reviewed, and the panels discussed these scores and determined a final "overall 
investment justification" score for each urban area. DHS told us it decided to give these two scores equal 
weights — 0.5 to the total of investment scores, and 0.5 to the overall investment justification score — and 
averaged them together to determine one final effectiveness score. While officials told us they discussed 
alternative weights, they did not have any data to indicate that they would be more appropriate than 
those chosen. 

At the end of the panel conference DHS used a survey to gather feedback, and 80 percent of the 102 peer 
reviewers responded and provided comments. The following include some of the preliminary survey 
results that DHS reported: 

• Eighty-three percent of survey respondents agreed or strongly agreed that the fiscal year 2006 HSGP 
peer review resulted in objective, consistent, and defensible scores and feedback. 

• Ninety-six percent of respondents agreed or strongly agreed that each panel included balanced 
representation from different regions, disciplines, and backgrounds. 

• Sixty-nine percent of respondents disagreed or strongly disagreed that the level of effort necessary 
for the review process was clearly communicated, and 78 percent disagreed or strongly disagreed 
that panelists were given sufficient time to review, score, and return scoring sheets to DHS prior to 
the panel convention. 

At the time of our review, DHS planned to continue to use a peer review process to assess effectiveness, 
but did not indicate whether it would be making changes to the process for fiscal year 2007. 



"DHS reported that a consensus on final scores was not required. Instead, reviewers' scores within each panel were averaged. 
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Appendix V: UASI Grant Allocation Approach for Fiscal Year 2006 



In fiscal year 2006, DHS used a new method to determine the amounts of UASI grants to each of the 46 
eligible urban areas, based primarily on the risk and effectiveness assessments, but final allocation 
decisions were made by the Secretary of Homeland Security. DHS reported that the aim of considering 
both factors — risk and effectiveness — is to allocate and apply HSGP resources to generate the highest 
return on investment and, as a result, to strengthen national preparedness. The risk and effectiveness 
scores did not automatically translate into funding amounts, but rather, according to DHS, the scores 
informed the decisions made by DHS officials. While all eligible urban areas that applied for UASI grants 
received funding, DHS set priorities to determine how much each urban area would receive. When 
making funding decisions, DHS prioritized those areas estimated to have the highest risk of a successful 
terrorist attack, while still rewarding those areas that offered effective ways to address homeland 
security needs. As a result, the risk assessment was given a greater weight than the effectiveness 
assessment when allocating funds. 

DHS Allocation Tool Used to Categorize Urban Areas and Set Priorities 

DHS established funding priorities before making allocation decisions. For example, DHS officials told 
us the Secretary of Homeland Security selected an approach that considered both the risk and 
effectiveness assessments when making allocation decisions, rather than using the outcomes of only one 
of the assessments. This approach combined the two assessments by using a graphical tool — a two-by- 
two matrix — to create four categories that would be used to set funding priorities (Figure 4 illustrates 
the two-by-two matrix used by DHS). The four funding categories were: Category I — higher risk, higher 
effectiveness; Category II — higher risk, lower effectiveness; Category III — lower risk, higher 
effectiveness; and Category IV — lower risk, lower effectiveness. 

To create these four categories, DHS made judgments that affected the category in which urban areas 
fell. For example, dividing lines were drawn on the horizontal axis for effectiveness scores and the 
vertical axis for risk scores to create the four categories. Specifically, DHS officials told us they 
calculated a "natural inflection point" among the risk rankings of the 46 eligible urban areas, thereby 
determining the dividing line on the risk axis. DHS reported that about a third of the urban areas were 
above the dividing line and therefore considered "higher risk" and about two-thirds were below the line 
and thus, "lower risk." DHS officials told us they selected the median of the effectiveness scores as the 
midpoint on the horizontal axis, and those areas to the right of this point were considered "higher 
effectiveness" and those to the left "lower effectiveness." Each of the 46 eligible urban areas was plotted 
into one of the following categories according to their combination of risk and effectiveness scores. 
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Figure 4: DHS Allocation Tool Used in Fiscal Year 2006 UASI Funding 




O Circles are sized relative 
to allocation amount 

I. H ig he r ri sk/h ig he r eft active n ess 

II. Higher risk/lower effectiveness 

III. Lower risk/higher effectiveness 

IV. Lower risk/lower effectiveness 

Source: GAO analysis of DHS documents and information provided in interviews. 



Determining the Final Allocations 

According to DHS, it considered many different distributions of funding to each of the four categories, 
and decided to give Category I the highest funding priority and Category IV the lowest funding priority. 
The figure above illustrates the funding priorities it reported making, in which each circle represents a 
hypothetical urban area and the size of the circle corresponds to the relative amounts of the grant 
awards (i.e., a larger circle indicates a greater allocation amount). DHS conducted what it described as 
an optimization process to produce many possible options of funding amounts to each category. DHS 
told us that once the allotments to categories were decided, DHS used a formula to determine the grant 
award for each urban area. DHS stated that it decided to prioritize the outcomes of the risk analysis over 
the effectiveness assessment, and so it made the policy decision to give each urban area's risk score a 
weight of 2/3 and the effectiveness score a weight of 1/3 when calculating the formula. DHS officials did 
not indicate whether or not they considered other weights for the risk and effectiveness scores. DHS 
reported that some stakeholders expressed frustration that the effectiveness assessment was not 
assigned a greater weight, since the peer review process required considerable time and effort. As was 
previously described in appendix IV, at the time urban areas were completing their applications, DHS 
had not yet determined the weights that would be applied to the risk and effectiveness scores. DHS 
officials expect that risk and effectiveness scores will both factor into allocation decisions for fiscal year 
2007, but they do not currently know whether or not the weights given to risk and effectiveness will 
change in fiscal year 2007. 



DHS officials told us they presented funding options to the Secretary of Homeland Security, who made 
the final decision about funding allocations. The official from the Office of Grants and Training we spoke 
to did not provide additional details about the information presented to the Secretary to inform his 
decision, and did not know what other goals or data may have factored into the allocation decision. DHS 
also reported that it determined the need to treat two urban areas differently than the other urban areas 
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when making funding decisions because it considered them to be outliers in the risk analysis. DHS 
officials told us these areas have consequences so great that they cannot be appropriately accounted for 
in the risk model. DHS did not specify what methods it used to determine the amount of these two UASI 
grants. 

All of the 46 eligible urban areas that applied for a fiscal year 2006 UASI grant received funding. DHS 
reported that 70 percent of UASI funding went to the higher-risk urban areas in Categories I and II of the 
two-by-two matrix, and 45 percent of available funding went to the five urban areas with the highest 
relative risk estimates. The total amount of UASI funds DHS allocated in fiscal year 2006 decreased by 
14 percent from fiscal 2005, but individual funding percentage changes varied among the 46 grantees. 
For example, among the 46 urban areas, fifteen experienced an increase in funding and 28 saw a funding 
decrease. Three of the 35 areas did not receive funding in fiscal year 2005, but were identified as eligible 
to apply for funding through the risk assessment in fiscal year 2006. The total amount awarded to these 
three urban areas was $23,620,000. The table below describes the allocations to each urban area in fiscal 
years 2005 and 2006, and illustrates the percentage change between years. 



Table 6: Percent Change in UASI Funding between Fiscal Year 2005 and Fiscal Year 2006 



Urban area 8 


Fiscal year 
2005 

allocation 


Fiscal year 
2006 

allocation 


Percent change in 
funds from fiscal year 
2005 to 2006 


Eligible areas through risk assessment 


New recipients in fiscal year 2006 


FL - Ft. Lauderdale Area 





$9,980,000 




FL - Orlando Area 





$9,440,000 




TN -Memphis Area 





$4,200,000 




Increased funding 


NJ -Jersey City/Newark Area 


$19,172,120 


$34,330,000 


79% 


NC -Charlotte Area 


$5,479,243 


$8,970,000 


64% 


GA -Atlanta Area 


$13,117,499 


$18,660,000 


42% 


Wl -Milwaukee Area 


$6,325,872 


$8,570,000 


35% 


FL - Jacksonville Area 


$6,882,493 


$9,270,000 


35% 


MO - St. Louis Area 


$7,040,739 


$9,200,000 


31% 


CA -Los Angeles/Long Beach Area 


$69,235,692 


$80,610,000 


16% 


IL - Chicago Area 


$45,000,000 


$52,260,000 


16% 


MO - Kansas City Area 


$8,213,126 


$9,240,000 


13% 


Ml - Detroit 


$17,068,580 


$18,630,000 


9% 


FL - Miami Area 


$15,828,322 


$15,980,000 


1% 


Reduced funding 


OR - Portland Area 


$10,391,037 


$9,360,000 


(10%) 


TX -Houston Area 


$18,570,464 


$16,670,000 


(10%) 


PA -Philadelphia Area 


$22,818,091 


$19,520,000 


(14%) 


MD - Baltimore 


$11,305,357 


$9,670,000 


(14%) 


CA -Bay Area 


$33,226,729 


$28,320,000 


(15%) 


OH - Cincinnati Area 


$5,866,214 


$4,660,000 


(21%) 


WA - Seattle Area 


$1 1 ,840,034 


$9,150,000 


(23%) 


IN - Indianapolis Area 


$5,664,822 


$4,370,000 


(23%) 


MN - Twin Cities Area 


$5,763,411 


$4,310,000 


(25%) 


TX -San Antonio Area 


$5,973,524 


$4,460,000 


(25%) 
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HI - Honolulu Area 


$6,454,763 


$4,760,000 


(26%) 


MA - Boston Area 


$26,000,000 


$18,210,000 


(30%) 


OH - Cleveland Area 


$7,385,100 


$4,730,000 


(36%) 


CA -Anaheim/Santa Ana Area 


$19,825,462 


$11,980,000 


(40%) 


DC -National Capital Region 


$77,500,000 


$46,470,000 


(40%) 


NY -New York City 


$207,563,21 1 


$124,450,000 


(40%) 


OH - Columbus Area 


$7,573,005 


$4,320,000 


(43%) 


TX - Dallas/Fort Worth/Arlington Area 


$24,355,870 


$13,830,000 


(43%) 


PA -Pittsburgh Area 


$9,635,991 


$4,870,000 


(49%) 


LA -New Orleans Area 


$9,305,180 


$4,690,000 


(50%) 


CO - Denver Area 


$8,718,395 


$4,380,000 


(50%) 


Total funding for 35 eligible areas 


$749,100,346 


$642,520,000 


(14%) 




Sustainment areas" 


Increased funding 


KY -Louisville Area 


$5,000,000 


$8,520,000 


70% 


NE -Omaha Area 


$5,148,300 


$8,330,000 


62% 


CA -Sacramento Area 


$6,085,663 


$7,390,000 


21% 


FL - Tampa Area 


$7,772,791 


$8,800,000 


13% 


Reduced funding 


NV -Las Vegas Area 


$8,456,728 


$7,750,000 


(8%) 


OK - Oklahoma City Area 


$5,570,181 


$4,102,000 


(26%) 


OH - Toledo Area 


$5,307,598 


$3,850,000 


(27%) 


LA -Baton Rouge Area 


$5,226,495 


$3,740,000 


(28%) 


CA -San Diego Area 


$14,784,191 


$7,990,000 


(46%) 


NY -Buffalo Area 


$7,207,995 


$3,710,000 


(49%) 


AZ -Phoenix Area 


$9,996,463 


$3,920,000 


(61%) 


Total funding for 11 sustainment 
areas 


$80,556,405 


$68,102,000 


(15%) 




Total UASI funding allocated to 46 
urban areas 


$829,656,751 


$710,622,000 


(14%) 



Source: GAO analysis. 
Notes: 

a. For a description of the cities, counties, and other geographic areas included in each urban area, see appendix II, table 3. 

b. Sustainment area: an urban area that received UASI funding in fiscal year 2005, but was not deemed eligible to apply 
through the fiscal year 2006 risk assessment. However, DHS extended eligibility to these areas for one additional grant 
cycle. 

DHS Actions after the Fiscal Year 2006 UASI Grants Were Awarded 

After DHS awarded the fiscal year 2006 UASI grants it took additional steps to provide information about 
the grant determination process and to gather feedback from stakeholders. These steps included 
providing award letters that summarized the risk and effectiveness assessments for each urban area, 
requiring states to conduct grant reporting activities, and hosting an HSGP after-action conference. 

• DHS provided individual award letters that included basic descriptions of the risk and 
effectiveness assessments. The award letter, which announced the amount of the urban area's 
fiscal year 2006 UASI award, also provided high-level feedback. For example, counts of asset 
information and geographic attributes DHS used to estimate relative risk were included in the 
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letter. It also described whether DHS's estimate of relative risk placed the urban area in the 
(1) top 25 percent, (2) top 50 percent, (3) bottom 50 percent, or (4) bottom 25 percent, compared 
to the other eligible urban areas. The letter did not provide the urban areas with their specific 
risk score or ranking, however. Summary information was also provided on the results of the 
effectiveness assessment, including which investments were anticipated to be the most and least 
effective. The award letter did not explain how the risk and effectiveness assessments were used 
by DHS to determine final grant allocation amounts. 

• Through its grant reporting process, DHS gathered additional information about how the 
fiscal year 2006 UASI grants were to be spent. Once it allocated the UASI funds, DHS allowed 
the recipient urban areas to decide how to spend the grant under some conditions with specific 
reporting requirements. la According to DHS's fiscal year 2006 grant guidance, grants were to be 
awarded to the respective State Administrative Agencies, which were required to notify DHS 
within 60 days of the award date as to how the grant funds were allocated. 16 DHS also reported 
that grant recipients would be monitored periodically to ensure that the program goals, 
objectives, timeliness, budgets, and other related program criteria were being met. Officials from 
DHS's Office of Grants and Training reported that DHS plans to ask grant recipients how they 
spent their fiscal year 2006 funds. DHS officials told us that they plan to consider information 
when making decisions for fiscal year 2008 UASI allocations. 

• DHS convened a Homeland Security Grant Program After-Action conference. At the July 2006 
conference DHS gathered feedback on the UASI grant award process. The conference held 
working groups on homeland security planning, the HSGP guidance and application, the risk 
assessment, and the effectiveness assessment. DHS officials told us that the conference provided 
a feedback loop intended to bolster stakeholder support and promote transparency. The state 
and local partners who participated in the working groups at the conference produced 32 
substantive recommendations to improve upon the HSGP process for fiscal year 2007 and 
beyond. For example, one of the risk assessment working group's recommendations was that 
DHS should provide detailed briefings to state and local partners on the core components of the 
risk methodology used in the fiscal year 2006 process as one step to improve the transparency of 
the risk analysis process. The effectiveness assessment working group recommended eliminating 
the overall investment justification score, as it believed it was not beneficial and was not a true 
representation of the quality of the application. DHS reported that state and local partners agreed 
the overall fiscal year 2006 planning process was the most effective and constructive thus far, 
and that the process helped to standardize the focus of state and local programs around key 
homeland security capabilities. 

According to DHS officials, stakeholder feedback on the fiscal year 2006 UASI grant process has been 
obtained and is being considered and incorporated into the fiscal year 2007 process where appropriate. 
DHS stated it will continue to regularly seek stakeholder input and feedback to ensure that state and 
local governments are fully informed and that the process proceeds in a collaborative fashion in fiscal 
year 2007. For example, DHS reported plans to convene stakeholder meetings to receive input on how to 
make specific grant programs more user-friendly and transparent, including a midterm review during the 
HSGP application process. 



'°DHS required that conditions established by peer reviewers be met before it funded an investment with a score below a certain 
threshold. 

'"Subsequent information on actual expenditures was to be reported every 6 months through the Biannual Strategy 
Implementation Report. 
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